Safety considerations for self-driving vehicles

ABSTRACT

The technology relates to detection of aberrant driving situations during operation of a vehicle in an autonomous driving mode. Aberrant situations may include potential theft or unsafe conditions, which are determined according to one or more signals. The signals are derived from information detected about the environment around the vehicle, such as from one or more sensors disposed on the vehicle. In response to an aberrant situation, the vehicle may take various corrective action, such as rerouting, locking down the vehicle or communicating with remote assistance. The type of corrective action taken may depend on a type of cargo being transported or whether one or more passengers are in the vehicle. If there are passengers, the system may communicate with the passengers via the passenger&#39;s client computing devices or by presenting visual or audible information via a user interface system of the vehicle.

BACKGROUND

Autonomous vehicles, such as vehicles that do not require a humandriver, can be used to aid in the transport of trailered (e.g., towed)cargo, such as freight, livestock or other items from one location toanother. Such vehicles may operate in a fully autonomous mode or apartially autonomous mode where a person may provide some driving input.Cargo theft has been a concern for conventional transportation with ahuman driver, and is expected to continue to be a concern in semi- orfully-autonomous transportation.

BRIEF SUMMARY

Aspects of the technology relate to detecting aberrant drivingsituations for a self-driving vehicle, such as a possible theft ofcargo. Once detected, the self-driving vehicle may determine options forcorrective action and select a particular action to take.

According to one aspect, a vehicle is configured to operate in anautonomous driving mode. Here, the vehicle comprises a driving system, aperception system, a communication system and a control system. Thedriving system includes a steering subsystem, an acceleration subsystemand a deceleration subsystem to control driving of the vehicle in theautonomous driving mode. The perception system includes one or moresensors configured to detect objects in an environment external to thevehicle. The communication system is configured to provide wirelessconnectivity with one or more remote devices. The control systemincludes one or more processors. The control system is operativelycoupled to the driving system, the communication system and theperception system. The control system is configured to receive sensordata from the perception system and obtained by the one or more sensors,and to evaluate the sensor data. The sensor data is evaluated todetermine whether an aberrant situation exists in the environmentexternal to the vehicle, including to check for one or more signalsindicative of an atypical behavior relative to an expected behavior forthe external environment. Upon detection of the aberrant situation, thecontrol system is configured perform at least one of transmitinformation about the aberrant situation to a remote assistance serviceusing the communication system, transmit information about the aberrantsituation to an emergency service, or take corrective action via thedriving system.

In one example, the corrective action includes maneuvering to avoid theaberrant situation in the autonomous driving mode. In another example,the corrective action includes maneuvering to avoid the aberrantsituation in accordance with information received from the remoteassistance service. The expected behavior may be eitherlocation-dependent or situation-dependent.

In one scenario, the check for one or more signals indicative of anatypical behavior relative to an expected behavior for the externalenvironment includes evaluation of a position or arrangement of at leastone person, other vehicle or debris in the external environment relativeto a current location of the vehicle. The information about the aberrantsituation may include one or more of a still image, Lidar data, a videoimage or an audio segment captured by a sensor of the perception system.

The vehicle may further comprise a secure data storage system inoperative communication with the control system. Here, the controlsystem is further configured to store the information about the aberrantsituation in the secure data storage system.

Upon detection of the aberrant situation, the control system may befurther configured to determine whether the aberrant situation is apermissible aberrant situation or an impermissible aberrant situation.In this case, the control system is configured to determine whether theaberrant situation is the permissible aberrant situation or animpermissible aberrant situation based on a confidence level of theaberrant situation.

In another example, the control system is configured to select aspecific type of corrective action based on a type of cargo beingtransported by the vehicle. In a further example, the vehicle is a cargovehicle. Here, the control system is configured to activate one or morelocation sensors on cargo upon detection of the aberrant situation.

And the vehicle may further comprise a user interface subsystemoperatively coupled to the control system. In this case, upon detectionof the aberrant situation, the control system is further configured toprovide status information to one or more passengers of the vehicle viathe user interface subsystem.

According to another aspect, a method of operating a vehicle in anautonomous driving mode is provided. The method comprises receiving, bya control system, sensor data from one or more sensors of a perceptionsystem of the vehicle; evaluating, by the control system, the receivedsensor data to determine whether an aberrant situation exists in anenvironment external to the vehicle, including checking for one or moresignals indicative of an atypical behavior relative to an expectedbehavior for the external environment; and upon detection of theaberrant situation, the control system performing at least one oftransmitting information about the aberrant situation to a remoteassistance service via a communication system of the vehicle,transmitting information about the aberrant situation to an emergencyservice, or taking corrective action via a driving system of thevehicle.

In one example, the corrective action includes maneuvering to avoid theaberrant situation in the autonomous driving mode. In another example,the corrective action includes maneuvering to avoid the aberrantsituation in accordance with information received from the remoteassistance service.

Checking for one or more signals indicative of an atypical behaviorrelative to an expected behavior for the external environment mayinclude evaluating a position or arrangement of at least one person,other vehicle or debris in the external environment relative to acurrent location of the vehicle.

Upon detection of the aberrant situation, the method may includedetermining whether the aberrant situation is a permissible aberrantsituation or an impermissible aberrant situation.

The method may further comprise selecting a specific type of correctiveaction based on a type of cargo being transported by the vehicle.

In one example, the vehicle is a cargo vehicle. Here the method furtherincludes activating one or more location sensors on cargo upon detectionof the aberrant situation.

And in another example, upon detection of the aberrant situation, themethod further comprises providing status information to one or morepassengers of the vehicle via a user interface subsystem of the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an example cargo vehicle arrangement for use withaspects of the technology.

FIG. 1B illustrates an example passenger vehicle arrangement for usewith aspects of the technology.

FIGS. 2A-B are functional diagrams of an example tractor-trailer vehiclein accordance with aspects of the disclosure.

FIG. 3 is a function diagram of an example passenger vehicle inaccordance with aspects of the disclosure.

FIGS. 4A-B illustrate example sensor fields of view for use with aspectsof the technology.

FIGS. 5A-C illustrate exemplary corrective driving scenarios inaccordance with aspects of the disclosure.

FIGS. 6A-B illustrate another corrective driving scenario in accordancewith aspects of the disclosure.

FIGS. 7A-B illustrate an exemplary assistance configuration inaccordance with aspects of the disclosure.

FIG. 8 illustrates an example method of operating a vehicle inaccordance with aspects of the technology.

DETAILED DESCRIPTION

The technology relates to a monitoring system for self-driving vehiclesthat is capable of detecting potential cargo thefts and other aberrantsituations. Such technology is form-factor agnostic, meaning that it isapplicable to autonomous trucks and other cargo vehicles, as well aspassenger vehicles. In conjunction with detecting aberrant situations,the system can redirect the self-driving vehicle along an alternateroute or otherwise take corrective action. Relevant authorities can bealerted as needed.

Example Vehicle Systems

FIG. 1A illustrates an example cargo vehicle 100, such as atractor-trailer truck, and FIG. 1B illustrates an example passengervehicle 150, such as a minivan. The cargo vehicle 100 may include, e.g.,a single, double or triple trailer, or may be another medium or heavyduty truck such as in commercial weight classes 4 through 8. As shown,the truck includes a tractor unit 102 and a single cargo unit or trailer104. The trailer 104 may be fully enclosed, open such as a flat bed, orpartially open depending on the freight or other type of cargo (e.g.,livestock) to be transported. The tractor unit 102 includes the engineand steering systems (not shown) and a cab 106 for a driver and anypassengers. In a fully autonomous arrangement, the cab 106 may not beequipped with seats or manual driving components, since no person may benecessary.

The trailer 104 includes a hitching point, known as a kingpin. Thekingpin is configured to pivotally attach to the tractor unit. Inparticular, the kingpin attaches to a trailer coupling, known as afifth-wheel, that is mounted rearward of the cab. Sensor units may bedeployed along the tractor unit 102 and/or the trailer 104. The sensorunits are used to detect information about the surroundings around thecargo vehicle 100. For instance, as shown the tractor unit 102 mayinclude a roof-mounted sensor assembly 106 and one or more side sensorassemblies 108, and the trailer 104 may employ one or more sensorassemblies 110, for example mounted on the left and/or right sides ofthe trailer 104. In some examples, the tractor unit 102 and trailer 104also may include other various sensors for obtaining information aboutthe tractor unit 102's and/or trailer 104's interior spaces.

Similarly, the passenger vehicle 150 may include various sensors forobtaining information about the vehicle's external environment. Forinstance, a roof-top housing 152 may include a Lidar sensor as well asvarious cameras and/or radar units. Housing 154, located at the frontend of vehicle 150, and housings 156 a, 156 b on the driver's andpassenger's sides of the vehicle may each incorporate a Lidar or othersensor. For example, housing 156 a may be located in front of thedriver's side door along a quarterpanel of the vehicle. As shown, thepassenger vehicle 150 also includes housings 158 a, 158 b for radarunits, Lidar and/or cameras also located towards the rear roof portionof the vehicle. Additional Lidar, radar units and/or cameras (not shown)may be located at other places along the vehicle 100. For instance,arrow 160 indicates that a sensor unit may be positioned along the readof the vehicle 150, such as on or adjacent to the bumper. In someexamples, the passenger vehicle 150 also may include various sensors forobtaining information about the vehicle 150's interior spaces.

While certain aspects of the disclosure may be particularly useful inconnection with specific types of vehicles, the vehicle may be any typeof vehicle including, but not limited to, cars, trucks, motorcycles,buses, recreational vehicles, etc.

FIG. 2A illustrates a block diagram 200 with various components andsystems of a cargo vehicle, such as a truck, farm equipment orconstruction equipment, configured to operate in a fully orsemi-autonomous mode of operation. By way of example, there aredifferent degrees of autonomy that may occur for a vehicle operating ina partially or fully autonomous driving mode. The U.S. National HighwayTraffic Safety Administration and the Society of Automotive Engineershave identified different levels to indicate how much, or how little,the vehicle controls the driving. For instance, Level 0 has noautomation and the driver makes all driving-related decisions. Thelowest semi-autonomous mode, Level 1, includes some drive assistancesuch as cruise control. Level 2 has partial automation of certaindriving operations, while Level 3 involves conditional automation thatcan enable a person in the driver's seat to take control as warranted.In contrast, Level 4 is a high automation level where the vehicle isable to drive without assistance in select conditions. And Level 5 is afully autonomous mode in which the vehicle is able to drive withoutassistance in all situations. The architectures, components, systems andmethods described herein can function in any of the semi orfully-autonomous modes, e.g., Levels 1-5, which are referred to hereinas “autonomous” driving modes. Thus, reference to an autonomous drivingmode includes both partial and full autonomy.

As shown in the block diagram of FIG. 2A, the vehicle includes a controlsystem of one or more computing devices, such as computing devices 202containing one or more processors 204, memory 206 and other componentstypically present in general purpose computing devices. The controlsystem may constitute an electronic control unit (ECU) of a tractor unitor other computing system of the vehicle. The memory 206 storesinformation accessible by the one or more processors 204, includinginstructions 208 and data 210 that may be executed or otherwise used bythe processor 204. The memory 206 may be of any type capable of storinginformation accessible by the processor, including a computingdevice-readable medium. The memory is a non-transitory medium such as ahard-drive, memory card, optical disk, solid-state, tape memory, or thelike. Systems may include different combinations of the foregoing,whereby different portions of the instructions and data are stored ondifferent types of media.

The instructions 208 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions” and “programs” may be used interchangeably herein.The instructions may be stored in object code format for directprocessing by the processor, or in any other computing device languageincluding scripts or collections of independent source code modules thatare interpreted on demand or compiled in advance. The data 210 may beretrieved, stored or modified by one or more processors 204 inaccordance with the instructions 208. In one example, some or all of thememory 206 may be an event data recorder or other secure data storagesystem configured to store vehicle diagnostics and/or detected sensordata, which may be on board the vehicle or remote, depending on theimplementation.

The one or more processor 204 may be any conventional processors, suchas commercially available CPUs. Alternatively, the one or moreprocessors may be a dedicated device such as an ASIC or otherhardware-based processor. Although FIG. 2A functionally illustrates theprocessor(s), memory, and other elements of computing devices 202 asbeing within the same block, such devices may actually include multipleprocessors, computing devices, or memories that may or may not be storedwithin the same physical housing. Similarly, the memory 206 may be ahard drive or other storage media located in a housing different fromthat of the processor(s) 204. Accordingly, references to a processor orcomputing device will be understood to include references to acollection of processors or computing devices or memories that may ormay not operate in parallel.

In one example, the computing devices 202 may form an autonomous drivingcomputing system incorporated into vehicle 100. The autonomous drivingcomputing system may capable of communicating with various components ofthe vehicle. For example, returning to FIG. 2A, the computing devices202 may be in communication with various systems of the vehicle,including a driving system including a deceleration system 212 (forcontrolling braking of the vehicle), acceleration system 214 (forcontrolling acceleration of the vehicle), steering system 216 (forcontrolling the orientation of the wheels and direction of the vehicle),signaling system 218 (for controlling turn signals), navigation system220 (for navigating the vehicle to a location or around objects) and apositioning system 222 (for determining the position of the vehicle).

The computing devices 202 are also operatively coupled to a perceptionsystem 224 (for detecting objects in the vehicle's environment), a powersystem 226 (for example, a battery and/or gas or diesel powered engine)and a transmission system 230 in order to control the movement, speed,etc., of the vehicle in accordance with the instructions 208 of memory206 in an autonomous driving mode which does not require or needcontinuous or periodic input from a passenger of the vehicle. Some orall of the wheels/tires 228 are coupled to the transmission system 230,and the computing devices 202 may be able to receive information abouttire pressure, balance and other factors that may impact driving in anautonomous mode.

The computing devices 202 may control the direction and speed of thevehicle by controlling various components. By way of example, computingdevices 202 may navigate the vehicle to a destination locationcompletely autonomously using data from the map information andnavigation system 220. Computing devices 202 may use the positioningsystem 222 to determine the vehicle's location and the perception system224 to detect and respond to objects when needed to reach the locationsafely. In order to do so, computing devices 202 may cause the vehicleto accelerate (e.g., by increasing fuel or other energy provided to theengine by acceleration system 214), decelerate (e.g., by decreasing thefuel supplied to the engine, changing gears, and/or by applying brakesby deceleration system 212), change direction (e.g., by turning thefront or other wheels of vehicle 100 by steering system 216), and signalsuch changes (e.g., by lighting turn signals of signaling system 218).Thus, the acceleration system 214 and deceleration system 212 may be apart of a drivetrain or other transmission system 230 that includesvarious components between an engine of the vehicle and the wheels ofthe vehicle. Again, by controlling these systems, computing devices 202may also control the transmission system 230 of the vehicle in order tomaneuver the vehicle autonomously.

As an example, computing devices 202 may interact with decelerationsystem 212 and acceleration system 214 in order to control the speed ofthe vehicle. Similarly, steering system 216 may be used by computingdevices 202 in order to control the direction of vehicle. For example,if the vehicle is configured for use on a road, such as atractor-trailer truck or a construction vehicle, the steering system 216may include components to control the angle of wheels of the tractorunit 102 to turn the vehicle. Signaling system 218 may be used bycomputing devices 202 in order to signal the vehicle's intent to otherdrivers or vehicles, for example, by lighting turn signals or brakelights when needed.

Navigation system 220 may be used by computing devices 202 in order todetermine and follow a route to a location. In this regard, thenavigation system 220 and/or memory 206 may store map information, e.g.,highly detailed maps that computing devices 202 can use to navigate orcontrol the vehicle. As an example, these maps may identify the shapeand elevation of roadways, lane markers, intersections, crosswalks,speed limits, traffic signal lights, buildings, signs, real time trafficinformation, vegetation, or other such objects and information. The lanemarkers may include features such as solid or broken double or singlelane lines, solid or broken lane lines, reflectors, etc. A given lanemay be associated with left and right lane lines or other lane markersthat define the boundary of the lane. Thus, most lanes may be bounded bya left edge of one lane line and a right edge of another lane line.

The perception system 224 also includes sensors for detecting objectsexternal to the vehicle. The detected objects may be other vehicles,obstacles in the roadway, traffic signals, signs, trees, etc. Forexample, the perception system 224 may include one or more lightdetection and ranging (Lidar) sensors, sonar devices, radar units,cameras (e.g., optical and/or infrared), inertial sensors (e.g.,gyroscopes or accelerometers), and/or any other detection devices thatrecord data which may be processed by computing devices 202. The sensorsof the perception system 224 may detect objects and theircharacteristics such as location, orientation, size, shape, type (forinstance, vehicle, pedestrian, bicyclist, etc.), heading, and speed ofmovement, etc. The raw data from the sensors and/or the aforementionedcharacteristics can sent for further processing to the computing devices202 periodically and continuously as it is generated by the perceptionsystem 224. Computing devices 202 may use the positioning system 222 todetermine the vehicle's location and perception system 224 to detect andrespond to objects when needed to reach the location safely. Inaddition, the computing devices 202 may perform calibration ofindividual sensors, all sensors in a particular sensor assembly, orbetween sensors in different sensor assemblies.

As indicated in FIG. 2A, the sensors of the perception system 224 may beincorporated into one or more sensor assemblies 232. In one example, thesensor assemblies 232 may be arranged as sensor towers integrated intothe side-view mirrors on the truck, farm equipment, constructionequipment or the like. Sensor assemblies 232 may also be positioned atdifferent locations on the tractor unit 102 or on the trailer 104 (seeFIG. 1A), or along different portions of passenger vehicle 150 (see FIG.1B). The computing devices 202 may communicate with the sensorassemblies located on both the tractor unit 102 and the trailer 104 ordistributed along the passenger vehicle 150. Each assembly may have oneor more types of sensors such as those described above.

Also shown in FIG. 2A is a communication system 234 and a couplingsystem 236 for connectivity between the tractor unit and the trailer.The coupling system 236 includes a fifth-wheel at the tractor unit and akingpin at the trailer. The communication system 234 may include one ormore wireless network connections to facilitate communication with othercomputing devices, such as passenger computing devices within thevehicle, and computing devices external to the vehicle, such as inanother nearby vehicle on the roadway or at a remote network. Thenetwork connections may include short range communication protocols suchas Bluetooth, Bluetooth low energy (LE), cellular connections, as wellas various configurations and protocols including the Internet, WorldWide Web, intranets, virtual private networks, wide area networks, localnetworks, private networks using communication protocols proprietary toone or more companies, Ethernet, WiFi and HTTP, and various combinationsof the foregoing.

FIG. 2B illustrates a block diagram 240 of an example trailer. As shown,the system includes an ECU 242 of one or more computing devices, such ascomputing devices containing one or more processors 244, memory 246 andother components typically present in general purpose computing devices.The memory 246 stores information accessible by the one or moreprocessors 244, including instructions 248 and data 250 that may beexecuted or otherwise used by the processor(s) 244. The descriptions ofthe processors, memory, instructions and data from FIG. 2A apply tothese elements of FIG. 2B.

The ECU 242 is configured to receive information and control signalsfrom the trailer unit. The on-board processors 244 of the ECU 242 maycommunicate with various systems of the trailer, including adeceleration system 252 (for controlling braking of the trailer),signaling system 254 (for controlling turn signals), and a positioningsystem 256 (for determining the position of the trailer). The ECU 242may also be operatively coupled to a perception system 258 (fordetecting objects in the trailer's environment) and a power system 260(for example, a battery power supply) to provide power to localcomponents. Some or all of the wheels/tires 262 of the trailer may becoupled to the deceleration system 252, and the processors 244 may beable to receive information about tire pressure, balance, wheel speedand other factors that may impact driving in an autonomous mode, and torelay that information to the processing system of the tractor unit. Thedeceleration system 252, signaling system 254, positioning system 256,perception system 258, power system 260 and wheels/tires 262 may operatein a manner such as described above with regard to FIG. 2A. Forinstance, the perception system 258, if employed as part of the trailer,may include at least one sensor assembly 264 having one or more Lidarsensors, sonar devices, radar units, cameras, inertial sensors, and/orany other detection devices that record data which may be processed bythe ECU 242 or by the processors 204 of the tractor unit.

The trailer also includes a set of landing gear 266, as well as acoupling system 268. The landing gear 266 provide a support structurefor the trailer when decoupled from the tractor unit. The couplingsystem 268, which may be a part of coupling system 236 of the tractorunit, provides connectivity between the trailer and the tractor unit.The coupling system 268 may include a connection section 270 to providebackward compatibility with legacy trailer units that may or may not becapable of operating in an autonomous mode. The coupling system includesa kingpin 272 configured for enhanced connectivity with the fifth-wheelof an autonomous-capable tractor unit.

FIG. 3 illustrates a block diagram 300 of various systems of a passengervehicle. As shown, the system includes one or more computing devices302, such as computing devices containing one or more processors 304,memory 306 and other components typically present in general purposecomputing devices. The memory 306 stores information accessible by theone or more processors 304, including instructions 308 and data 310 thatmay be executed or otherwise used by the processor(s) 304. Thedescriptions of the processors, memory, instructions and data from FIG.2A apply to these elements of FIG. 3.

As with the computing devices 202 of FIG. 2A, the computing devices 302of FIG. 3 may control computing devices of an autonomous drivingcomputing system or incorporated into a passenger vehicle. Theautonomous driving computing system may be capable of communicating withvarious components of the vehicle in order to control the movement ofthe passenger vehicle according to primary vehicle control code ofmemory 306. For example, computing devices 302 may be in communicationwith various, such as deceleration system 312, acceleration system 314,steering system 316, signaling system 318, navigation system 320,positioning system 322, perception system 324, power system 326 (e.g.,the vehicle's engine or motor), transmission system 330 in order tocontrol the movement, speed, etc. of the in accordance with theinstructions 208 of memory 306. The wheels/tires 328 may be controlleddirectly by the computing devices 302 or indirectly via these othersystems. These components and subsystems may operate as described abovewith regard to FIG. 2A. For instance, the perception system 324 alsoincludes one or more sensors 332 for detecting objects external to thevehicle. The sensors 332 may be incorporated into one or more sensorassemblies as discussed above.

Computing devices 202 may include all of the components normally used inconnection with a computing device such as the processor and memorydescribed above as well as a user interface subsystem 334. The userinterface subsystem 334 may include one or more user inputs 336 (e.g., amouse, keyboard, touch screen and/or microphone) and various electronicdisplays 338 (e.g., a monitor having a screen or any other electricaldevice that is operable to display information). In this regard, aninternal electronic display may be located within a cabin of thepassenger vehicle (not shown) and may be used by computing devices 302to provide information to passengers within the vehicle. Output devices,such as speaker(s) 340 may also be located within the passenger vehicle.

The passenger vehicle also includes a communication system 342, whichmay be similar to the communication system 234 of FIG. 2A. For instance,the communication system 342 may also include one or more wirelessnetwork connections to facilitate communication with other computingdevices, such as passenger computing devices within the vehicle, andcomputing devices external to the vehicle, such as in another nearbyvehicle on the roadway, or a remote server system. The networkconnections may include short range communication protocols such asBluetooth, Bluetooth low energy (LE), cellular connections, as well asvarious configurations and protocols including the Internet, World WideWeb, intranets, virtual private networks, wide area networks, localnetworks, private networks using communication protocols proprietary toone or more companies, Ethernet, WiFi and HTTP, and various combinationsof the foregoing.

Example Implementations

In view of the structures and configurations described above andillustrated in the figures, various implementations will now bedescribed.

The vehicle is configured to detect aberrant situations, which mayinclude, by way of example, possible cargo theft. This may be done byanalyzing sensor data received from the various sensors around thevehicle, to determine, for instance, a specific situation or that one ormore factors indicate that the situation is aberrant. By way of example,the sensors may detect an unexpected road blockage or pedestrians at alocation (e.g., on a highway) where they would not typically be seen.These and other situations are described in detail below.

In order to detect the environment and conditions around the vehicle,different types of sensors and layouts may be employed. Examples ofthese were discussed above with regard to FIGS. 1A and 1B. The field ofview for each sensor can depend on the sensor placement on a particularvehicle. In one scenario, the information from one or more differentkinds of sensors may be employed so that the tractor-trailer orpassenger vehicle may operate in an autonomous mode. Each sensor mayhave a different range, resolution and/or field of view (FOV).

For instance, the sensors may include a long range FOV Lidar and a shortrange FOV Lidar. In one example, the long range Lidar may have a rangeexceeding 50-250 meters, while the short range Lidar has a range nogreater than 1-50 meters. Alternatively, the short range Lidar maygenerally cover up to 10-15 meters from the vehicle while the long rangeLidar may cover a range exceeding 100 meters. In another example, thelong range is between 10-200 meters, while the short range has a rangeof 0-20 meters. In a further example, the long range exceeds 80 meterswhile the short range is below 50 meters. Intermediate ranges ofbetween, e.g., 10-100 meters can be covered by one or both of the longrange and short range Lidars, or by a medium range Lidar that may alsobe included in the sensor system. In addition to or in place of theseLidars, a set of cameras (e.g., optical and/or infrared) may bearranged, for instance to provide forward, side and rear-facing imagery.Similarly, a set of radar sensors may also be arranged to provideforward, side and rear-facing data.

FIGS. 4A-B illustrate example sensor configurations and fields of viewon a cargo vehicle. In particular, FIG. 4A presents one configuration400 of Lidar, camera and radar sensors. In this figure, one or moreLidar units may be located in sensor housing 402. In particular, sensorhousings 402 may be located on either side of the tractor unit cab, forinstance integrated into a side view mirror assembly. In one scenario,long range Lidars may be located along a top or upper area of the sensorhousings 402. For instance, this portion of the housing 402 may belocated closest to the top of the truck cab or roof of the vehicle. Thisplacement allows the long range Lidar to see over the hood of thevehicle. And short range Lidars may be located along a bottom area ofthe sensor housings 402, closer to the ground, and opposite the longrange Lidars in the housings. This allows the short range Lidars tocover areas immediately adjacent to the cab. This would allow theperception system to determine whether an object such as anothervehicle, pedestrian, bicyclist, etc., is next to the front of thevehicle and take that information into account when determining how todrive or turn in view of an aberrant condition.

As illustrated in FIG. 4A, the long range Lidars on the left and rightsides of the tractor unit have fields of view 404. These encompasssignificant areas along the sides and front of the vehicle. As shown,there is an overlap region 406 of their fields of view in front of thevehicle. A space is shown between regions 404 and 406 for clarity;however in actuality there would desirably be overlapping coverage. Theshort range Lidars on the left and right sides have smaller fields ofview 408. The overlap region 406 provides the perception system withadditional or information about a very important region that is directlyin front of the tractor unit. This redundancy also has a safety aspect.Should one of the long range Lidar sensors suffer degradation inperformance, the redundancy would still allow for operation in anautonomous mode.

FIG. 4B illustrates coverage 410 for either (or both) of radar andcamera sensors on both sides of a tractor-trailer. Here, there may bemultiple radar and/or camera sensors in each of the sensor housings 412.As shown, there may be sensors with side and rear fields of view 414 andsensors with forward facing fields of view 416. The sensors may bearranged so that the side and rear fields of view 414 overlap, and theside fields of view may overlap with the forward facing fields of view416. As with the long range Lidars discussed above, the forward facingfields of view 416 also have an overlap region 418. This overlap regionprovides similar redundancy to the overlap region 406, and has the samebenefits should one sensor suffer degradation in performance.

While not illustrated in FIGS. 4A-4B, other sensors may be positioned indifferent locations to obtain information regarding other areas aroundthe vehicle, such as along the rear or underneath the vehicle.

Example Scenarios

For situations in which the self-driving vehicle is fully autonomouswithout a driver being present, it is important that the system not onlydetect what is going on in the external environment, but also evaluatethe environment to determine whether an aberrant situation exists. Thiscan be done by checking for common signals that are indicative ofatypical behavior, e.g., from people or other vehicles.

One particularly relevant scenario involves cargo transportation. Here,a fully autonomous truck may be traveling on highways or other roadswith expensive, fragile or otherwise precious cargo. Theft of cargo haslong been a significant problem in the trucking industry, and that trendmay be expected to continue with driverless trucks. In order to addressthis, the system is capable of detecting the truck's externalenvironment with various sensors such as Lidar, radar, cameras, etc., asdiscussed above. For instance, the sensors can detect a road blockage,e.g., where all lanes are blocked or there is not enough room for thetruck to maneuver around the blockage.

The on-board system can then check for signals that would indicate apotential theft scenario. For example, the system may analyze whetherpeople are present in a place where they would not be expected, such asin the lanes of a highway. Other signals could include the orientationor positioning of other vehicles or other objects blocking the road orotherwise impeding the ability of vehicles to maneuver. The system canuse the signals to differentiate a permissible situation (e.g., policecar blocking traffic due to an accident) from other situations, e.g.,theft or safety.

Other types of relevant signals can include the following. Unexpected orunusual road signs, such as a stop sign or a traffic light on a freeway.The system can evaluate from its own maps or other data, or frominformation received from a fleet or central depot, whether the roadsign should or should not be present.

In addition to people being present on the highway, another signal couldbe a person (or persons) very close to the self-driving vehicle. Afurther signal could be an indication that a lock on the trailer isbroken. This may be detected by external sensors, such as an opticalcamera or Lidar, or by a sensor integrated with the lock. The systemcould also recognize police or other emergency vehicles and determinewhether a stopped vehicle is of that type of vehicle. Yet another signalmay be multiple tires losing pressure at the same time. This could becompared against visual sensor data of the road surface. Multipledepressurizations could indicate that it wasn't an accidental tirepuncture; rather it may have occurred due to a tire blowout strip laidon the roadway.

Once the potential theft situation is detected, the system can operateto reduce the likelihood of theft and take other action. For example,the self-driving truck may contact a centralized security team, a remoteassistant or law enforcement about the situation. Real-time sensorinformation, such as live camera feeds from the truck, can be wirelesslytransmitted via one or more communication channels back to a centraloffice or to law enforcement.

There may be times when the aberrant situation is detected and the truckis far away from law enforcement or other assistance. In this case, thesensors (e.g., sensor assemblies 232 or 264 of FIGS. 2A-B, or sensorassemblies 332 of FIG. 3) may record still images, video images or audioto document the situation. By way of example, one or more cameras areable to record still or video images, which may be optical or infraredimages. Images or other sensor data from Lidar or radar sensors may alsobe gathered, in addition to or in lieu of camera images. And microphonesor other transducers may collect audio information. This information maybe transmitted to a remote system (e.g., cloud-based storage) forbackup. The data may also be stored locally in a black-box recorder.Tracking devices on cargo pallets or other items may be activated.

If possible, the truck or other self-driving vehicle may take correctivedriving action. For instance, as shown in example 500 of FIG. 5A, ifsome or all of the driving lanes 502 are blocked on the highway but theshoulder is open or an off-ramp 504 is available, the truck can re-routeto maneuver around or otherwise avoid the blockage, for instance bytaking the off-ramp as shown by dashed line 506. FIG. 5B illustratesanother example 510 of a self-driving truck traveling along anon-highway route. Here, the planned route 512 may be straight, but thisroute is shown as blocked or otherwise impeded by vehicles. Thus, asseen in FIG. 5C, the truck may reroute along another path 522 to avoidthe blockage. In some examples, the rerouting discussed herein mayinvolve backing up, traveling off-road, or other unusual maneuvers. Inthese cases, the location and/or positioning of the other vehicles onthe roadway may be a signal of aberrant behavior, such as an attempt tocause the truck to stop, and not necessarily an accident or othersituation.

When remote assistance is available, corrective driving action may bedone under the direction of, or by, a person at the remote assistancelocation. Here, for example, the truck may not typically be authorizedto drive autonomously on the shoulder of the highway or on a non-roadsurface; however, the remote assistance personnel may maneuver the truckremotely or provide such authorization. Thus, in one example the remoteassistance personnel may provide an alternative path for the vehicle tofollow. In another example, the remote assistance personnel mayauthorize the vehicle to follow a proposed path planned by the vehicle.In still a further example, the remote assistance personnel maytemporarily take over one or more driving operations, for instance untilthe aberrant situation has been addressed.

An alternative type of corrective action is to automatically deflate oneor more tires of the vehicle. Another example of corrective action wouldbe to lower the landing gear of the trailer to prevent the trailer fromeasily being moved. A further corrective action would be to engage alocking feature between the fifth wheel and kingpin to make it difficultto unhitch the trailer from the truck. Other corrective actions mayinclude locking the vehicle's steering wheel, disabling the gas pedaland/or other pedals, disabling a gear shifting mechanism, etc. Any orall of these types of corrective action could be employed so that thevehicle cannot easily be driven away if a potential theft scenario isdetected

Another factor to take into account with regard to trucking is the typeof cargo that is being transported. For example, is it a fluid or solid,how is the weight distributed in the trailer unit, what is the value ofthe cargo, is it perishable, etc.?

For certain kinds of cargo, the nature of response by the vehicle couldchange in accordance with the type of cargo. By way of example, if thetruck is carrying sensitive cargo, it might take specific correctiveactions when a confidence level of the type of scenario may be low(e.g., on the order of 50%), whereas for other cargo such correctiveactions would only be taken if there was a higher (e.g., 75% or higher)confidence level as to the type of scenario.

As noted above, the technology is form-factor agnostic. Similarapproaches can be applied to passenger vehicles such as autonomous taxisor buses. When the system detects and determines that the situation isatypical, various steps can be taken. For instance, the vehicle doorsmay be locked (or remain locked). The windows could be automaticallytinted to prevent someone outside the vehicle from seeing into thevehicle. A detour or re-route around the blockage may be taken ifpossible. The system may proactively inform the authorities and/or othervehicles, for instance if the self-driving vehicle is part of a fleet.And passengers within the vehicle can be alerted or otherwise informedof the situation and the corrective action being taken.

FIGS. 6A and 6B illustrate one example of corrective action for aself-driving passenger vehicle. As shown in view 600 of FIG. 6A, theself-driving vehicle may have a planned route 602 indicated by thedashed line. However, another vehicle (or person, debris, etc.) mayblock, occlude or otherwise impede the planned route 602. Thus, as shownin FIG. 6B, the self-driving vehicle may determine a new route 612 toavoid the impediment.

As discussed above, the on-board system may communicate with remoteassistance, law enforcement (e.g., police) or other emergency services,and/or with passengers of the vehicle. One example of this is shown inFIGS. 7A and 7B. In particular, FIGS. 7A and 7B are pictorial andfunctional diagrams, respectively, of an example system 700 thatincludes a plurality of computing devices 702, 704, 706, 708 and astorage system 710 connected via a network 460. System 700 also includesvehicles 712 and 714, which may be configured the same as or similarlyto vehicles 100 and 150 of FIGS. 1A and 1B. Vehicle 712 and/or vehicles714 may be part of a fleet of vehicles. Also shown is an emergencyservices vehicle 715, e.g., a police car. Although only a few vehiclesand computing devices are depicted for simplicity, a typical system mayinclude significantly more.

As shown in FIG. 7B, each of computing devices 702, 704, 706 and 708 mayinclude one or more processors, memory, data and instructions. Suchprocessors, memories, data and instructions may be configured similarlyto the ones described above with regard to FIG. 2A.

The various computing devices and vehicles may communication via one ormore networks, such as network 716. The network 716, and interveningnodes, may include various configurations and protocols including shortrange communication protocols such as Bluetooth, Bluetooth LE, theInternet, World Wide Web, intranets, virtual private networks, wide areanetworks, local networks, private networks using communication protocolsproprietary to one or more companies, Ethernet, WiFi and HTTP, andvarious combinations of the foregoing. Such communication may befacilitated by any device capable of transmitting data to and from othercomputing devices, such as modems and wireless interfaces.

In one example, computing device 702 may include one or more servercomputing devices having a plurality of computing devices, e.g., a loadbalanced server farm, that exchange information with different nodes ofa network for the purpose of receiving, processing and transmitting thedata to and from other computing devices. For instance, computing device702 may include one or more server computing devices that are capable ofcommunicating with the computing devices of vehicles 712 and/or 714, aswell as computing devices 704, 706 and 708 via the network 716. Forexample, vehicles 712 and/or 714 may be a part of a fleet of vehiclesthat can be dispatched by a server computing device to variouslocations. In this regard, the computing device 702 may function as adispatching server computing system which can be used to dispatchvehicles to different locations in order to pick up and drop offpassengers or to pick up and deliver cargo. The vehicles 715 may alsocommunicate with the dispatching server computing device. In addition,vehicles 712, 714 and/or 715 may also directly or indirectly with othervehicles 712, 714 and/or 715. In addition, server computing device 702may use network 716 to transmit and present information to a user of oneof the other computing devices or a passenger of a vehicle. In thisregard, computing devices 704, 706 and 708 may be considered clientcomputing devices.

As shown in FIG. 7A each client computing device 704, 706 and 708 may bea personal computing device intended for use by a respective user 718,and have all of the components normally used in connection with apersonal computing device including a one or more processors (e.g., acentral processing unit (CPU)), memory (e.g., RAM and internal harddrives) storing data and instructions, a display (e.g., a monitor havinga screen, a touch-screen, a projector, a television, or other devicesuch as a smart watch display that is operable to display information),and user input devices (e.g., a mouse, keyboard, touchscreen ormicrophone). The client computing devices may also include a camera forrecording video streams, speakers, a network interface device, and allof the components used for connecting these elements to one another.

Although the client computing devices may each comprise a full-sizedpersonal computing device, they may alternatively comprise mobilecomputing devices capable of wirelessly exchanging data with a serverover a network such as the Internet. By way of example only, clientcomputing devices 706 and 708 may mobile phones or devices such as awireless-enabled PDA, a tablet PC, a wearable computing device (e.g., asmartwatch), or a netbook that is capable of obtaining information viathe Internet or other networks.

In some examples, client computing device 704 may be a remote assistanceworkstation used by an administrator or operator to communicate withpassengers as discussed further below. Although only a single remoteassistance workstation 704 is shown in FIGS. 7A-7B, any number of suchwork stations may be included in a given system. Moreover, althoughoperations work station is depicted as a desktop-type computer,operations works stations may include various types of personalcomputing devices such as laptops, netbooks, tablet computers, etc.

Storage system 710 can be of any type of computerized storage capable ofstoring information accessible by the server computing devices 702, suchas a hard-drive, memory card, ROM, RAM, DVD, CD-ROM, flash drive and/ortape drive. In addition, storage system 710 may include a distributedstorage system where data is stored on a plurality of different storagedevices which may be physically located at the same or differentgeographic locations. Storage system 710 may be connected to thecomputing devices via the network 716 as shown in FIGS. 7A-7B, and/ormay be directly connected to or incorporated into any of the computingdevices.

Storage system 710 may store various types of information. For instance,the storage system 710 may also store autonomous vehicle controlsoftware which is to be used by vehicles, such as vehicles 712 or 714,to operate such vehicles in an autonomous driving mode. Storage system710 may store various types of information as described in more detailbelow. This information may be retrieved or otherwise accessed by aserver computing device, such as one or more server computing devices702, in order to perform some or all of the features described herein.

For instance, storage system 710 may store log data. This log data mayinclude, for instance, sensor data generated by a perception system,such as the perception systems of vehicles 712 or 714. As an example,the sensor data may include raw sensor data as well as data identifyingdefining characteristics of perceived objects (including other roadusers) such as shape, location, orientation, speed, etc. of objects suchas vehicles, pedestrians, bicyclists, vegetation, curbs, lane lines,sidewalks, crosswalks, buildings, etc. The log data may also include“event” data identifying different types of events such as collisions ornear collisions with other objects, planned trajectories describing aplanned geometry and/or speed for a potential path of the vehicle,actual locations of the vehicles at different times, actualorientations/headings of the vehicle at different times, actual speeds,accelerations and decelerations of the vehicle at different times,classifications of and responses to perceived objects, behaviorpredictions of perceived objects, status of various systems (such asacceleration, deceleration, perception, steering, signaling, routing,power, etc.) of the vehicle at different times including logged errors,inputs to and outputs of the various systems of the vehicle at differenttimes, etc. As such, these events and the sensor data may be used to“recreate” the vehicle's environment, including perceived objects, andbehavior of a vehicle in a simulation. The sensor data may also be usedby the server 702 to detect anomalous or otherwise aberrant situations,such as potential theft situations for cargo vehicle 712.

At least some of this log data may be “ride data,” that is, log datagenerated during a particular trip or ride taken by a passenger in anautonomous vehicle such as vehicle 714. Ride data may include all of theaforementioned features of the log data or may include only specifictypes of data such as motion planning commands from a vehicle's plannersystem, telemetry from the vehicle, context from the map informationused to control the vehicle, processed or raw sensor data for other roadusers (such as vehicles, bicyclists, pedestrians, etc.) from thevehicle's perception system, acceleration information, jerk (or thederivative of acceleration) information, etc. Ride data may also beassociated with ride feedback provided by one or more passengers for aparticular ride

As discussed above, the self-driving vehicle may communication withremote assistance in order to handle aberrant situations. For instance,should the vehicle determine that it has encountered an aberrantsituation, it may send a query and/or data to remote assistance. Thequery may include a request for an updated route, authorization tomodify the current route, additional information regarding the aberrantsituation, etc. The data may include raw and/or processed sensor data,vehicle log data and the like. For instance, it may include one or morestill images, video and/or an audio segment(s).

In conjunction or alternatively to communicating with remote assistance,the vehicle may communicate with law enforcement (e.g., police) or otheremergency services. This may be occur, for example, when a determinationhas been made from various signal information that a cargo vehicle hasencountered a potential theft situation or unsafe situation whenpassengers are in the vehicle. Various responses or actions may be takenas discussed above, including securing the vehicle and/or takingcorrective driving action, for instance which modifies or deviates froma planned route.

In a situation where there are passengers, the vehicle or remoteassistance may communicate directly or indirectly with the passengers'client computing device. Here, for example, information may be providedto the passengers regarding the current situation, actions being takenor to be taken in response to the situation, etc.

FIG. 8 illustrates an example 800 of a method of operating a vehicle inan autonomous driving mode in view of the above. For instance, as shownin block 802, sensor data is received from the vehicle's perceptionsystem. In block 804, the sensor data is evaluated to determine whetheran aberrant situation exists. Per block 806, this may involve checkingfor one or more signals indicative of an atypical behavior relative toan expected behavior. For example, are one or more cars or othervehicles stopped on the roadway in a manner that would indicate apotential theft situation rather than traffic or an accident. Then, asshown in block 808, in view of this, performing at least one of (i)transmitting information about the aberrant situation to a remoteassistance service, or (ii) taking corrective driving action. The lattermay include, e.g., altering a planned route, pulling over, exiting aroadway, locking down the vehicle, etc.

Unless otherwise stated, the foregoing alternative examples are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the embodiments should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples; rather, the examples are intended to illustrate onlyone of many possible embodiments. Further, the same reference numbers indifferent drawings can identify the same or similar elements. Theprocesses or other operations may be performed in a different order orsimultaneously, unless expressly indicated otherwise herein.

The invention claimed is:
 1. A vehicle configured to operate in anautonomous driving mode, the vehicle comprising: a driving systemincluding a steering subsystem, an acceleration subsystem and adeceleration subsystem to control driving of the vehicle in theautonomous driving mode; a perception system including one or moresensors configured to detect objects in an environment external to thevehicle; a communication system configured to provide wirelessconnectivity with one or more remote devices; and a control systemincluding one or more processors, the control system operatively coupledto the driving system, the communication system and the perceptionsystem, the control system being configured to: receive sensor data fromthe perception system and obtained by the one or more sensors; evaluatethe sensor data to determine whether an aberrant situation exists in theenvironment external to the vehicle, including to check for one or moresignals indicative of an atypical behavior relative to an expectedbehavior for the external environment; and upon detection of theaberrant situation, take corrective action via the driving system. 2.The vehicle of claim 1, wherein the corrective action includesmaneuvering to avoid the aberrant situation in the autonomous drivingmode.
 3. The vehicle of claim 1, wherein the corrective action includesmaneuvering to avoid the aberrant situation in accordance withinformation received from a remote assistance service.
 4. The vehicle ofclaim 1, wherein the expected behavior is either location-dependent orsituation-dependent.
 5. The vehicle of claim 1, wherein the check forone or more signals indicative of an atypical behavior relative to anexpected behavior for the external environment includes evaluation of aposition or arrangement of at least one person, other vehicle or debrisin the external environment relative to a current location of thevehicle.
 6. The vehicle of claim 1, wherein information about theaberrant situation includes one or more of a still image, Lidar data, avideo image or an audio segment captured by a sensor of the one or moresensors of the perception system.
 7. The vehicle of claim 1, furthercomprising a secure data storage system in operative communication withthe control system, wherein the control system is further configured tostore information about the aberrant situation in the secure datastorage system.
 8. The vehicle of claim 1, wherein upon detection of theaberrant situation, the control system is further configured todetermine whether the aberrant situation is a permissible aberrantsituation or an impermissible aberrant situation.
 9. The vehicle ofclaim 8, wherein the control system is configured to determine whetherthe aberrant situation is the permissible aberrant situation or animpermissible aberrant situation based on a confidence level of theaberrant situation.
 10. The vehicle of claim 1, wherein the controlsystem is configured to select a specific type of corrective actionbased on a type of cargo being transported by the vehicle.
 11. Thevehicle of claim 1, wherein the vehicle is a cargo vehicle, and thecontrol system is configured to activate one or more location sensors oncargo upon detection of the aberrant situation.
 12. The vehicle of claim1, further comprising a user interface subsystem operatively coupled tothe control system, wherein upon detection of the aberrant situation,the control system is further configured to provide status informationto one or more passengers of the vehicle via the user interfacesubsystem.
 13. The vehicle of claim 1, wherein upon detection of theaberrant situation, the control system is further configured to performat lest one of transmit information about the aberrant situation to aremote assistance service using the communication system or transmitinformation about the aberrant situation to an emergency service.
 14. Amethod of operating a vehicle in an autonomous driving mode, the methodcomprising: receiving, by a control system, sensor data from one or moresensors of a perception system of the vehicle; evaluating, by thecontrol system, the received sensor data to determine whether anaberrant situation exists in an environment external to the vehicle,including checking for one or more signals indicative of an atypicalbehavior relative to an expected behavior for the external environment;and upon detection of the aberrant situation, the control system takingcorrective action via a driving system of the vehicle.
 15. The method ofclaim 14, wherein the corrective action includes maneuvering to avoidthe aberrant situation in the autonomous driving mode.
 16. The method ofclaim 14, wherein the corrective action includes maneuvering to avoidthe aberrant situation in accordance with information received from aremote assistance service.
 17. The method of claim 14, wherein thechecking for one or more signals indicative of an atypical behaviorrelative to an expected behavior or the external environment includesevaluating a position or arrangement of at least one person, othervehicle or debris in the external environment relative to a currentlocation of the vehicle.
 18. The method of claim 14, wherein upondetection of the aberrant situation, the method includes determiningwhether the aberrant situation is a permissable aberrant situation or animpermissible aberrant situation.
 19. The method of claim 14, furthercomprising selecting a specific type of corrective action based on atype of cargo being transported by the vehicle.
 20. The method of claim14, wherein the vehicle is a cargo vehicle, and the method includesactivating one or more location sensors on cargo upon detection of theaberrant situation.
 21. The method of claim 14, wherein upon detectionof the aberrant situation the method further comprises providing statusinformation to one or more passengers of the vehicle via a userinterface subsystem of the vehicle.
 22. The vehicle of claim 14, whereinupon detection of the one aberrant situation, the method furtherincludes the control system performing at least one of transmittinginformation about the aberrant situation to a remote assistance servicevia a communication system of the vehicle or transmitting informationabout the aberrant situation to an emergency service.